Run 5 produced a genuinely good review — accurate, specific, verified against the diff — and then mislabelled it. Five real findings, every one marked advisory, verdict clean. So a PR with five things to fix got agent:clean, which is the label a maintainer skims. Copilot read the same PR as needing amends, and Copilot was right.

Two causes, both in the rubric.

1. Confidence and severity were conflated

The old text put "anything you are less than confident about" in the advisory bucket. Those are orthogonal axes — a finding can be high-impact and uncertain, and that wording forced everything uncertain into the low bucket regardless of how much it mattered.

Severity is now rated by impact alone. Uncertainty goes in the detail text ("I could not verify whether X handles Y"), and there is a separate instruction: if you cannot assert a finding at all, do not file it. A hedged real finding beats a silent one; a confident wrong one is worse than both.

2. Binary severity forced the bad verdict

With only blocking / advisory, findings that individually do not block a merge collapse into "clean". Added a middle level:

The worked example for important is the case run 5 actually found: a test that asserts a failure path which silently succeeds in some environments, so it passes while proving nothing. A green suite makes that more dangerous, not less — which is precisely why it belongs above nit.

There is also a calibration note: if you find yourself marking everything a nit, you are under-calling. Would a careful human reviewer ask for a change before approving? "The maintainer could merge anyway" is true of almost everything and is not the test.

The verdict is now derived, not trusted

The workflow computes it from the findings instead of taking the model's word:

• any blocking → needs-work
• any findings at all → comments
• none → clean

A mismatch between the model's claim and the derived value is logged as a workflow notice, so drift is visible rather than silent.

Three labels, mutually exclusive; the other two are removed unconditionally so a /re-review cannot leave a stale pair. The heading now carries the tally — comments — 2 important, 3 nits — so it cannot disagree with the table underneath it.

Verified

Replaying run 5's actual findings through the new logic:

model claimed: clean
derived:       comments — 2 important, 3 nits
label:         agent:comments  (removes agent:clean, agent:needs-work)

Honest caveat

The verdict derivation is deterministic and will hold. The important-vs-nit split still depends on the model applying the new guidance, and I assigned those severities by hand in the replay above. Whether it actually calls the root-permissions test important rather than nit is the thing to watch on the next real run.

Second commit: issue resolution and whether the PR was warranted

Added to this PR rather than a separate one — it touches the same two files and would have conflicted.

Two gaps

The rubric checked scope creep — did the PR do more than the issue asked — but never completeness: did it do all of it. And it never asked whether the change should exist at all.

Neither was answerable, because the agent was never given the issue. Linked issues are now fetched with full body, title, state and labels, and included in the context. Only GraphQL exposes closingIssuesReferences; gh pr view --json does not have the field in gh 2.69.

Issue resolution

A partial fix on a PR that says "Closes #N" is important at minimum. Merging it closes the issue, and whatever is left over is silently lost — nobody returns to a closed issue.

It is asked to name which parts of a multi-part issue remain, and to say so when the issue is vague enough that several readings are defensible rather than inventing a standard the author could not have known.

Warranted

Maintainer attention is the scarcest resource in this project, and an unnecessary PR costs it whether or not the code is correct. The check asks: is there a real problem, does the codebase already do this, is it churn against working code, does it fight the direction set in CLAUDE.md.

This one needed guardrails and has them. It is the most discouraging thing the bot can say to a contributor, so the rubric states explicitly:

• small PRs from first-time contributors are how people start
• "this is small" is not a reason to reject anything
• an undiscussed but justified change is a conversation, not a rejection
• do not file this finding at all unless the case is clear

Coverage

43 of 66 open nanocoder PRs link an issue, so the resolution check has real coverage — and the 23 without are exactly the population the warranted check exists for.